Search for: All records

We propose a method for learning the posture and struc- ture of agents from unlabelled behavioral videos. Start- ing from the observation that behaving agents are gener- ally the main sources of movement in behavioral videos, our method, Behavioral Keypoint Discovery (B-KinD), uses an encoder-decoder architecture with a geometric bottle- neck to reconstruct the spatiotemporal difference between video frames. By focusing only on regions of movement, our approach works directly on input videos without requir- ing manual annotations. Experiments on a variety of agent types (mouse, fly, human, jellyfish, and trees) demonstrate the generality of our approach and reveal that our dis- covered keypoints represent semantically meaningful body parts, which achieve state-of-the-art performance on key- point regression among self-supervised methods. Addition- ally, B-KinD achieve comparable performance to supervised keypoints on downstream tasks, such as behavior clas- sification, suggesting that our method can dramatically re- duce model training costs vis-a-vis supervised methods.